Surgical tracking and procedural map analysis tool

ABSTRACT

In some embodiments, methods and systems are provided for accessing a surgical dataset including surgical data collected during performance of a surgical procedure. The surgical data can include video data of the surgical procedure. Using the surgical data, a plurality of procedural states associated with the surgical procedure can be determined. For a procedural state of the plurality of procedural states, temporal information can be identified that identifies a part of the video data to be associated with the procedural state. For the procedural state of the plurality of procedural states, electronic data can be generated that characterizes the part of the video data and outputting the electronic data associated with the plurality of procedural states.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. Nonprovisional applicationSer. No. 15/791,663, filed on Oct. 24, 2017, which is a continuation ofU.S. Nonprovisional application Ser. No. 15/495,705, filed on Apr. 24,2017, which claims the benefit of and the priority to U.S. ProvisionalApplication Number 62/464,606, filed on Feb. 28, 2017. Each of theseapplications is hereby incorporated by reference in its entirety for allpurposes.

FIELD

Methods and systems disclosed herein relate generally to providingprocedural guidance data during a surgical procedure. More specifically,procedural guidance data associated with a surgical procedure isrepresented by a pre-coded surgical procedural map that includes preciseand accurate coded descriptions of patient states and surgical actions.Real-time data is processed to identify a particular position within themap, and corresponding procedural guidance data is identified andavailed.

BACKGROUND

Variations in procedural methods between different surgeons toaccomplish a particular surgical procedure can influence proceduraloutcomes. Procedural consistency of individual surgeons, in addition tovariations in procedural methods, can also influence proceduraloutcomes. Presently, a number of surgeons achieve procedural outcomesthat are below a minimum acceptable level. Further, there are surgeonsthat achieve outcomes above the minimum but below an average proceduraloutcome. A tool is needed to help standardize procedural methods betweendifferent surgeons and assist individual surgeons with improvingprocedural consistency.

SUMMARY

In some embodiments, methods and systems are provided for accessing asurgical dataset including surgical data collected during performance ofa surgical procedure. The surgical data can include video data of thesurgical procedure. Using the surgical data, a plurality of proceduralstates associated with the surgical procedure can be determined. For aprocedural state of the plurality of procedural states, temporalinformation can be identified that identifies a part of the video datato be associated with the procedural state. For the procedural state ofthe plurality of procedural states, electronic data can be generatedthat characterizes the part of the video data and outputting theelectronic data associated with the plurality of procedural states.

Some embodiments of the present disclosure include a system includingone or more data processors. In some embodiments, the system includes anon-transitory computer readable storage medium containing instructionswhich when executed on the one or more data processors, cause the one ormore data processors to perform part of all of one or more methodsand/or part or all of one or more processes disclosed herein. Someembodiments of the present disclosure include a computer-program producttangibly embodied in a non-transitory machine-readable storage mediumincluding instructions configured to cause one or more data processorsto perform part of all of one or more methods and/or part or all of oneor more processes disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIG. 1 shows a block diagram of an embodiment of structures andtechnologies configured to provide surgical guidance using a surgicaldata structure;

FIG. 2 shows an embodiment of a surgical data structure;

FIG. 3 shows an alternate embodiment of a surgical data structure;

FIG. 4 shows relation of multiple surgical data structures;

FIG. 5 shows an embodiment of the data flow in the structures andtechnologies configured to provide surgical guidance using a surgicaldata structure;

FIGS. 6A, 6B, and 6C show illustrations of the presentation of actionguidance corresponding to the surgical data structure;

FIG. 7 shows a block diagram of an embodiment of a local serverconfigured to provide surgical guidance using a surgical data structure;

FIG. 8 shows a flowchart of a method for the automated provisioning ofreal-time custom procedural surgical guidance;

FIG. 9 shows a flowchart of a method for selecting a target node;

FIG. 10 shows a flowchart of a method for analyzing surgical data andoutputting electronic data associated with procedural states; and

FIG. 11 shows a flowchart of a method for updating a surgical datastructure.

DETAILED DESCRIPTION

The ensuing description provides preferred exemplary embodiment(s) only,and is not intended to limit the scope, applicability or configurationof the disclosure. Rather, the ensuing description of the preferredexemplary embodiment(s) will provide those skilled in the art with anenabling description for implementing a preferred exemplary embodiment.It is understood that various changes may be made in the function andarrangement of elements without departing from the spirit and scope asset forth in the appended claims.

The structures and technologies described herein may functionindependently. In some embodiments, the structures and technologiesdescribed herein may function in combination to process live data toidentify a real-time stage of a surgery, represented in a surgical datastructure, which can be used to retrieve or generate pertinentinformation (e.g., corresponding to a current or subsequent action) foran entire operating room. The structures and technologies describedherein may be integrated with existing operating room systems. Further,the structures and technologies described herein may include intelligentsystem that collects and analyzes operating room data and takes actionsthat maximize the chances of a desired procedural outcome.Fundamentally, the structures and technologies described herein use asurgical data structure as a reference during live surgical proceduressuch that appropriate guidance may be provided to the surgeon at theright moment of surgery. Structures and technologies described hereinmay describe an appropriate surgical route using the surgical datastructure and a number of inputs including one or more user inputs,operating room video or audio streams, smart surgical tools,accelerometer data from a surgeon or other operating room personnel, andpatient data. Current textual descriptions of surgery are often verbose,yet semantically constrained by the ambiguity of language they employ.Like GPS navigation relieved users of determining their position using amap or atlas, the structures and technologies described herein relievesurgeons and operating room personnel of tracking patient or proceduralstates and determining surgery progress from a reference. The structuresand technologies described herein determine the procedural location andprovide context to operating room personnel for the procedural locationand provisions appropriate procedural guidance.

In some embodiments, systems and methods used to process live data toidentify a real-time stage of a surgery, represented in a surgical datastructure, which can be used to retrieve or generate pertinentinformation (e.g., corresponding to a current or subsequent action). Asurgical data structure, that represents various stage in a surgery andcorresponding information, can be locally retrieved or received. Duringa surgery, live data can be collected (e.g., from a sensor at a wearabledevice, smart surgical instrument, etc.). The live surgical data can beprocessed in accordance with the surgical data structure associated withthe surgery to identify a specific procedural state (or stage).Procedural metadata associated with the state (e.g., which can includeguidance information associated with an anticipated and/or recommendednext procedural state) can be retrieved (e.g., from the surgical datastructure or from a location identified by the surgical data structure.)and presented.

Accessing the surgical data structure can include (for example)retrieving the data structure from a remote or local data store and/orreceiving the data structure from another device (e.g., a cloud server)in response to a request. The surgical data structure may include aplurality of nodes and a plurality of edges. Each edge of the pluralityof edges may be configured to connect two nodes of the plurality ofnodes. The nodes and edges may be associated with and/or arranged in anorder that corresponds to a sequence in which various actions may beperformed and/or states may be reached in a surgical procedure. Each ofone more or all nodes in the surgical data structure and/or each of one,more or all edges in the surgical data structure may be associated withprocedural metadata. The metadata can include description of and/orinformation corresponding an associated procedural state and/or surgicalaction. The surgical data structure may include a four dimensionalsurgical map, three dimensions corresponding to anatomy and the fourthdimension identifying a stage within a procedure.

Live surgical data may be collected by any device integrated with thesurgical navigation system. In some embodiments, live surgical data canbe collected at a wearable device worn by one or more surgical teammembers. Live surgical data can also be collected during a surgicalprocedure via (for example) captured images or video feeds, audio feeds,smart surgical tools, accelerometers on a wearable device on a surgeon,and/or team member inputs. The live surgical data may be encrypted, asnecessary, in order to protect the privacy of individually identifiablehealth information in compliance with the Health Insurance Portabilityand Accountability Act of 1996.

Third, a processing device may receive the live surgical data. Theprocessing device can be located in or near a surgery location or in thecloud. The processing device can be configured to identify theprocedural state and procedural metadata associated with the proceduralstate. The live surgical data can be processed (e.g., using featureextraction) to identify a state of the patient/surgery. The state of thepatient/surgery may be grounded in a physiological state (i.e., thecondition or state or the patient's body or bodily functions). In someembodiments, a discrete procedural state (i.e., procedural locationand/or procedural action) can be identified based on the state of thepatient/surgery and procedural metadata corresponding to a node in thesurgical data structure.

Metadata corresponding to the procedural state can be identified. Themetadata may be (for example) included in the surgical data structureitself and/or identified by a part of the surgical data structure. Forexample, a node may include data that identifies surgical tools for anupcoming surgical action or it may point to a location in a table,array, data structure, data store, and so on that may include theparticular information. The metadata can be used to generate real-timeguidance information for output (e.g., identifying a surgical action tobe taken, including an instruction to take a picture for documentation,identifying a tool to be used or prepped, presenting a vital sign formonitoring, paging for assistance, contacting a control center near anoperating room with an assistance request, etc.). Guidance informationcan correspond to decision-tree type of structure. The guidanceinformation may be configured to be presented as text data (e.g., to bepresented on a display) and/or non-text data (e.g., audio signals, imageoverlays, artificial reality data). The type of output (e.g.,information presentation versus control-center contact) may differdepending on (for example) a procedural state, patient state or action.

In some embodiments, live surgical data (or a processed version thereof,such as timestamped state identification) can be preserved for apost-hoc analysis. To improve the security corresponding to the livedata, the data may be stored so as to anonymize the data and/or toobscure and/or encrypt particular (e.g., private or identifying)features. Live surgical data may be processed to identify currentprocedural states throughout the surgery. Edges corresponding to actionsperformed to transition between consecutive states can further beinferred, detected, or identified via an input or other data. Thus, foreach surgery, trajectory data can identify a series of actions performedby a surgeon. This data may be aggregated, correlated with surgeryoutcomes and/or compared to guidance to, for example: (1) assessindividual surgeons' compliance with guidelines or requirements; (2)generate audit logs; (3) update recommended actions/edges and/orguidance; and/or (4) identify overall preferences with regard toactions.

Further, live data that includes video content can be used to update asurgical data structure. For instance, a determination that particularframes of video of a surgery corresponds to step 4 of a carpal tunnelprocedure can be used to annotate the video. The step identification canindicate what objects should be present in the video (e.g., a patient'shand, a surgeon's hand, and a scalpel blade with a handle masked by1+fingers). This knowledge can be used to automatically detect what ascalpel looks like with 1+fingers covering the handle, which can be usedto annotate the video.

Some techniques described herein can improve procedural consistencyacross entities involved in performance of a surgery (e.g., a surgeon,surgical team, hospital, etc.), improve procedural consistency for agiven entity involved in performance of a surgery and/or improveoutcomes of a surgery. Further, the real-time feedback can serve tosupplement or partly replace training provided by a physically orremotely present procedural expert. Thus, the techniques can promoteefficiency in this regard. Additionally, the feedback is provided at acritical time, such that mistakes can be avoided before any harm iscause to a patient.

Referring first to FIG. 1, an embodiment of a system 100 for processinglive data to identify a real-time stage of a surgery, represented in asurgical data structure, which can be used to retrieve or generatepertinent information is illustrated. The system 100 can collect livedata from a number of sources including (for example) a surgeon mountedheadset 110, a first additional headset 120, a second additional headset122, surgical data 150 associated with a patient 112, an operating roomcamera 134, an operating room microphone 136, and additional operatingroom tools not illustrated in FIG. 1. The live data is transmitted to awireless hub 160 in communication with a local server 170. The localserver 170 receives the live data from the wireless hub 160 over aconnection 162 and a surgical data structure from a remote server 180.The local server 170 can process the live data to identify a state ofthe patient/surgery. The local server 170 can identify a node in thesurgical data structure corresponding to the discrete procedural state.The local server 170 can process the metadata corresponding to thediscrete procedural state and generate real time guidance informationfor output to the appropriate devices in the operating room 102.

The local server 170 will be in contact with and synced with a remoteserver 180. In some embodiments the remote server 180 can be located inthe cloud 106. In some embodiments processing performed by the localserver 170 may be performed by the remote server 180. A global bank ofsurgical procedures, described using surgical data structures, may bestored at the remote server 180. Therefore, for any given surgicalprocedure, there is the option of running the system 100 as a local, orcloud based system. The local server 170 can create a surgical datasetthat records data collected during the performance of a surgicalprocedure. The local server 170 can analyze the surgical dataset orforward the surgical dataset to the remote server 180 upon thecompletion of the procedure for inclusion in a global surgical dataset.In some embodiments, the local server can anonymize the surgicaldataset. The proposed system 100 integrates data from the surgical datastructure and sorts guidance data appropriately in the operating roomusing additional components.

In certain embodiments, surgical guidance, retrieved from the surgicaldata structure, may include more information than necessary to assistthe surgeon with situational awareness. The system 100 may determinethat the additional operating room information may be more pertinent toother members of the operating room and transmit the information to theappropriate team members. Therefore, in certain embodiments, thecomplete system 100 provides surgical guidance to more components thanthe surgeon mounted headset 110.

In the illustrated embodiment, wearable devices such as a firstadditional headset 120 and a second additional headset 122 are includedin the system 100. Other members of the operating room team may benefitfrom receiving information and surgical guidance derived from thesurgical data structure on the wearable devices. For example, a surgicalnurse wearing the first additional headset 120 may benefit from guidancerelated to procedural steps and possible equipment needed for impendingsteps, see FIG. 6B. An anesthetist wearing the second additional headset122 may benefit from seeing the patient vital signs in the field ofview. In addition, the anesthetist may be the most appropriate user toreceive the real-time risk indication as one member of the operatingroom slightly removed from surgical action.

Distributing the real-time guidance information to team members canavoid work overload of an individual team member. For instance, asurgeon, wearing the surgeon's headset 110, may be overloaded ifpresented with an equipment list to prepare for the next step in thesurgical procedure. Therefore, to distribute workload, the systemincludes the provision of the first additional headset 120 and thesecond additional headset 122 to other members of the operating roomteam. The information presented to the nurse and the anesthetistillustrates how additional information included within the surgical datastructure may guide other members of the operating room team besides thesurgeon. The additional information and the integration of proceduralguidance for operating room team members can benefit the execution ofthe medical procedure.

In addition to the first additional headset 120 and the secondadditional headset 122 there may be any number of peripheral informationcommunicating devices including conventional displays 130, transparentdisplays that may be held between the surgeon and patient, ambientlighting 132, one or more operating room cameras 134, one or moreoperating room microphones 136, speakers 140 and procedural stepnotification screens placed outside the operating room to alert entrantsof critical steps taking place. These peripheral components can functionto provide information that may become critical during the procedure,yet the information displayed does not warrant constant distraction ofthe surgeon or another member of the operating room team.

Along with these peripheral communication devices, the system 100 can beintegrated with any devices within the operating room providing livesurgical data 150 concerning the state of the patient 112, for example,heart rate and blood pressure monitors. In addition to vital signs, thelive surgical data may include operating room characteristics such aslighting used, ambient temperatures, personnel identification/movement,and any other data that may indicate the progress of a surgicalprocedure. Sensors that can collect and provide live surgical data 150may include (for example) electromagnetic sensors, (e.g. hall effectsensors to determine when surgical tools are lifted off an operatingroom table), optical sensors, in addition to video sensors, and nearinfrared sensors (to assist with mapping patient anatomy, tracking bloodflow, etc.).

This fully integrated operating room system provides a structure throughwhich to automatically coordinate operating room devices with procedurallocation. The wireless hub 160 may use one or more communicationsnetworks to communicate with operating room devices including variouswireless protocols, such as IrDA, Bluetooth, Zigbee, Ultra-Wideband,and/or Wi-Fi. In some embodiments, existing operating room devices canbe integrated with the system 100. To illustrate, once a specificprocedural location is reached automatic functions can be set to prepareor change the state of relevant and appropriate medical devices toassist with impending surgical steps. For example, operating roomlighting 132 can be integrated into the system 100 and adjusted based onimpending surgical steps.

In another embodiment, in addition to procedural guidance, warnings ofhazards can be included within the system. The system 100 may usespeaker 140 to warn the operating room of radioactive tracer excretionfrom the patient 112 and subsequent collection into a suction bag. Avisual warning can be presented to operating room personnel using (forexample) the surgeon headset 110, the additional headsets, theconventional display 130, and/or operating room lighting 132.

In some embodiments, the system 100 may include a centralized hospitalcontrol center 172. This center may be connected to one, more or allactive surgeries and coordinate actions in critical situations as alevel-headed, but skilled, bystander. In addition to having access toall live data from each operating room (including; video stream,real-time surgical risk, procedural step and other data), a user (e.g.,healthcare professional) in the control center 172 may have access todata such as patient records and external personnel availability. Insome embodiments, the data available to control center personnel couldbe packaged and provided as educational material to (for example)medical trainees, residents, and/or students. Control center personnelmay have access to the surgical data structure using a surgical datastructure explorer. The control center personnel may update the surgicaldata structure during surgery based on changing patient conditions.

In the event of a surgical complication, the control center manager maybe able to directly page the most appropriate specialist on anintegrated specialist pager 174 to mitigate the situation. In additionto identifying solutions to critical situations before complications canfully develop, the control center manager has access to information noteasily integrated into the system 100 that may be requested by theoperating room at times of use. The control center 172 has access to allthe data within the operating room 102 and current procedural location,communication between the operating room 102 and control center 172 maytake many forms including text communications and direct audio or visuallinks.

In some embodiments, a risk indication can be displayed within thesystem 100 to inform the progression and seriousness of surgery. Tominimize distraction or stress, the system 100 may display the risksolely to a member of the operating room team removed from the presentstep of the surgery using (for example) the first additional headset 120or the second additional headset 122.

In certain embodiments, the indication of real-time surgical risk can beused to inform the operating room team of the risk or to triggercomplication avoidance methods by the system. For example, when the riskreaches a pre-defined threshold or a level computed in real-time, thesystem may provide additional guidance directly to the surgeon using thesurgeon headset 110, an alert communication can be transmitted a localhospital control center 172 so the situation can be monitored moreclosely, and/or a relevant specialist pager 174 can be activated tocontact a specialist necessary to resolve the situation. The risk levelcomputed in real-time may be based on the surgical map and/or theprobability of achieving optimal outcomes. In some embodiments, thesystem may dynamically flag an alert for procedural states, actions, orany other points in the surgical map.

In some embodiments, the system 100 calculates real-time surgical risk.The response of the system 100 based on this real-time risk can beflexible. The response may be adjusted by (for example) each hospital104 or control center 172. For example, the defined risk threshold uponwhich responses are actioned may be changed, or the individual parameterweighting for calculating risk may be manipulated. A user (e.g.,surgeon, nurse or team member) may access the surgical data structurepre-surgery in the control center 172 using a surgical data structureexplorer. The user may customize the risk thresholds or individualparameter weightings based on (for example) surgeon, team member,patient, and/or hospital specific characteristics.

The system 100 may adjust response actions based on other information,such as a surgeon's level of skill. The surgical data structure mayinclude a variable corresponding to a surgeon's level of skill (e.g.,corresponding to a particular occupational position, duration at aparticular facility, duration practicing surgeries generally or of aparticular type, complication probability based on past surgeries, andso on). The system 100 may determine the value for the variablecorresponding to the surgeon's level of skill by analyzing pre-recordedsurgical data associated with a surgeon or the variable may be enteredby a user (e.g., surgeon, nurse or team member). For example, a traineesurgeon may have a lower threshold of risk before actions or insistentwarnings are issued, a more experienced consultant may have to reach ahigher level of risk before the same actions are issued. Using a tieredrisk management system, the system 100 may reduce unnecessary risk byproviding guidance to a user (e.g., surgeon, nurse or team member) andnotifying specialists at the appropriate times. Reducing the unnecessaryrisk may decrease the occurrence of unfavorable patient outcomes.

A system 100 is not limited to the embodiments illustrated in FIG. 1.Some embodiments of a system 100 may remain the same with significantchange in component organization. The embodiments herein relate to theprovision of an intelligent system, which integrates with the wholeoperating room team, operating room data and procedural guidance; theembodiments herein do not relate to a specific medical device networkstructure.

Referring next to FIG. 2, one possible embodiment of a surgical datastructure 200 is shown. In some implementations, the system forprocessing live data to identify a real-time stage of a surgery, whichcan be used to retrieve or generate pertinent information, is groundedon a surgical data structure 200. The surgical data structure 200 allowsconcise descriptions of complex medical procedures. A discrete routethrough a surgical data structure 200 embodies a codified description ofa specific surgical procedure. The surgical data structure 200 mayinclude the information needed for the reproduction of an individualprocedure and additional information to describe both subtly differentand alternative procedural methods between individual users (e.g.,surgeon, nurse or team member), yet, no more information than necessaryto achieve both objectives.

In some embodiments, the precise route taken through a surgery ischaracterized by a series of transitions between discrete patientstates. The surgical data structure 200 may describe medical proceduresin full using graph theory or network structures. Within this graphicalnotation, a node 210 can represent a discrete physiological andprocedural state of the patient during a procedure (e.g., initialpreparation and sterilization complete, skin incision made, boneexposed, etc.). The node 210 may represent a pose, anatomical state,sterility, etc. The node 210 can be grounded in a 3D anatomical state. A3D anatomical state characterizes a state of a patient, such as patientposition, location of an organ, location of a surgical instrument,location of an incision, etc.

The node 210 and the associated discrete procedural state can beassociated with a set of procedural metadata 212 that can include (forexample) a description of and/or information corresponding to aprocedural state. Procedural metadata 212 associated with a node 210 caninclude, for example, a description of a surgical action to be taken, aninstruction to take a picture for documentation, a description of asurgical tool to be used or prepped, a description of a vital sign formonitoring; and/or a recommendation to page a control center forassistance. In some embodiments, the procedural metadata 212 can includea decision tree in machine readable format for use by a processingdevice or in natural language form for use by one or more system users.

An edge 240 can represent one or more surgical actions executed totransition between different nodes. In other embodiments, as shown inFIG. 3 and described further below, edges may represent distinct statesof the patient during a procedure and nodes may represent surgicalactions taken to transition between different edges. Each edge mayrepresent a medical action to be performed that will result in atransition between states. Each edge can be associated with informationabout the surgical action, such as an identification of the action, atime typically associated with the action, tools used for the action, aposition of interactions relative to anatomy, etc.

In some implementations, for each individual procedure, the surgicaldata structure 200 is a multidimensional space representing the manyroutes that a single procedure may take. A single route through thismultidimensional map may represent one method of executing the surgicalprocedure. The surgical data structure 200 may include a plurality ofroutes through the multidimensional map, including a preferred route230, indicated by a dashed line. The preferred route 230 may be linkedto a number of factors including (for example) user (e.g., surgeon,nurse or team member) experience, operating room equipment, operatingroom personnel, minimizing risk, and/or surgical outcomes. The surgicaldata structure 200 allows the precise definition of important surgicallandmarks using a constrained coded vocabulary.

Each node 210 may be associated with a set of procedural metadata 212that can include a description of and/or information corresponding tothe discrete procedural state. The description of and/or informationcorresponding to the discrete procedural state includes many differentattributes that describe the condition of a patient at a given pointduring a medical procedure. The attributes describing patient state mayinclude, but are certainly not limited to pose, anatomical state, andsterility. Any number of attributes may be needed in order to ensure anode is accurately described, and this may vary for different surgicalprocedures. The attributes associated with a discrete procedural statemay also include variables not related directly to the patient but tothe surgical procedure. For example, the personnel present in theoperating room may relate to the state of the surgery but not to anyparticular patient state. In some embodiments, the fewest number ofattributes is desirable. The description of and/or informationcorresponding to a discrete procedural state may describe a proceduralstate with sufficient detail to be used by a processing device in afeature extraction algorithm.

In some embodiments, each of one, more or all nodes is grounded in a 3Danatomical state (e.g., characterizing a state of a patient, such as aposition, location of an organ, location of a surgical instrument,location of an incision, and so on). As such, the node 210 may include adetailed description of anatomical state. Surgical interventions aim tomodify anatomy and their progress is constrained by patient topology.Thus, anatomical state and organization may present importantcheckpoints to split a complex procedure into discrete steps. Thesesteps must be passed, regardless of method or prior event sequence. Forexample, to perform a total knee replacement, the skin must be cut andthe bones at the knee joint exposed in order for the replacement knee tobe applied.

In certain embodiments, using a surgical data structure 200 based on agraph model of surgery, edges 240 describe the transition betweendifferent nodes—each node representing a different patient state. Assuch, edges 240 represent actions taken by the users (e.g., surgeon,nurse or team member) during the procedure. Each edge may be associatedwith action specific metadata 214, including, but not limited to: timetaken, tools used, state of the tools used, and position of interactionsrelative to anatomy. In other embodiments, nodes may represent actionstaken by the users (e.g., surgeon, nurse or team member) during theprocedure and edges may provide a description of the patient state.

Node-to-node transitions in surgery may take multiple different routesbetween the same two destinations depending on factors entirely distinctfrom the two nodes. In some implementations, node-aware systems andmethods facilitate flexible routes between a single destination node.Furthermore, different node-to-node transitions (i.e., surgicalactions), or entirely different routes through the surgical datastructure 200, may be recommended for different age patients in the sameprocedure.

In some embodiments, for the provision of the structures andtechnologies described herein, the surgical data structure 200 must betranslated from a generic coded anatomical model to real anatomy. Forexample, the surgical data structure 200 may need to be anatomicallyadjusted for success in specific procedures. Some embodiments may adjustthe surgical data structure 200 for obese patients, patients withatypical anatomy or patients who may have undergone significant surgicalprocedures in the past.

The surgical data structure 200 may include anatomical models that areboth malleable and include topologies that can be expected to remainconsistent between many different patients. In some embodiments, thesurgical data structure 200 may account for the anatomical distinctionbetween the sexes, which may impact the description of certainsurgeries. In certain embodiments, a selection of patient specificanatomic models may need to be implemented, or existing models adjustedin certain situations.

To further illustrate the use of a surgical data structure 200, anembodiment describing a subset of steps performed during the surgicalprocedure for carpal tunnel release is provided herein. Carpal tunnelrelease surgery is a simple procedure to relieve the symptoms of carpaltunnel syndrome. The carpel tunnel is a narrow passage within the wristcapped by the carpal ligament. The syndrome is caused when the mediannerve becomes compressed, often by swelling, within the carpal tunnel.The crucial step of carpal tunnel release surgery is to cut thisligament to release the median nerve and alleviate the symptoms ofcarpal tunnel syndrome. Above the ligament is a layer of tissue and alayer of skin respectively. Both must be divided in order to reach thecarpal ligament.

FIG. 2 details a simplified representation of a surgical data structure200 for a carpal tunnel release procedure. To accomplish the carpaltunnel release procedure, at least four crucial procedural nodes 210,216, 218, and 220 may be defined by the anatomic states of four relevantpieces of anatomy: the skin, wrist tissue, carpal ligament and mediannerve. Representative procedural metadata 212 is provided for each ofthe four crucial procedural nodes 210, 216, 218, and 220. During thecarpal tunnel release procedure the skin, tissue and ligament may beintact or divided and the nerve trapped or released. The proceduralmetadata 212 may include additional data not illustrated in FIG. 2. Theadditional data may include attributes used to define more nodes alongthe route of surgery (such as the sterility of the surgical site), butthese four anatomical states represent nodes through which the proceduremust pass to progress.

The four key procedural nodes 210, 216, 218, and 220 are connected byedges, which represent surgical interactions related to the associatednode-to-node transition. For example, edge 240 may represent the actionnecessary to transition from the first key procedural node 210 to thesecond key procedural node 216. The procedural metadata associated withthe second key procedural node 216 indicates that the skin is incised.In order to facilitate the transition between procedural states, edge240 represents the procedural action associated with the node-to-nodetransition. In this example, the action specific metadata 214 associatedwith edge 240 includes a description of the action, “incise the skin,”and the tools required, “scalpel.” The procedural action associated withedge 240 causes the associated physiological state of the skin totransition from intact to incised between node 210 and node 216respectively.

The four key procedural nodes 210, 216, 218, and 220 are illustrated onthe preferred route 230. The preferred route 230 passes through anintermediate node 232 to the third key procedural state 218. The set ofprocedural metadata associated with the third procedural state 218indicates that the tissue above the ligament has been divided. The finalkey procedural node 220 is associated with procedural metadata thatindicates that the nerve has been released and the surgical preferredroute 230 has reached a successful surgical outcome. The preferred routeincludes a final edge 228 that includes an instruction to “suture theskin” and a set of tools required to complete the procedural action.

In some embodiments, the surgical data structure 200 can account forsurgical procedures that do not reach a procedural state associated witha successful surgical outcome. FIG. 2 shows unsuccessful node 224 andits associated unsuccessful metadata 226 indicating that the nerve isstill trapped after traversing the nodes in the surgical data structure200.

For simplicity, the complete list of possible attributes associated withthe metadata for each node and edge has not been included in thisstylized representation. A longer list of possible attributes mayinclude, (for example) surface sterility, the interaction with specificinstruments or physical characteristics such as the pressure or tensionon specific anatomical structures, and/or patient vital signs.

To simplify the surgical data structure 200 of this surgery further, theroute may be represented as distinct nodes without any metadata 244. Thefour key procedural nodes 210, 216, 218, and 220 and relevant edgesshown in FIG. 2 are only a small subsection (individual surgical route)of a large, multidimensional surgical data structure including manypossible nodes and edges. A surgical data structure may represent allprojected surgical paths possible within the procedure.

In some embodiments, nodes and edges may be defined as a list ofcontingent events. For example, the retraction of the skin is contingenton its initial incision. By defining each set of data (i.e., a node oran edge) in contingent terms, the network is able to self-construct, byforming all possible actions given the extended rules of contingency. Incertain embodiments, the structures and technologies described hereinmay capture live surgical data to build descriptions of procedures,coded in the structure defined above. In some examples, the structuresand technologies described herein may harvest nodes and edges directlyfrom surgical procedures themselves and construct a surgical datastructure automatically.

In other embodiments, different successful routes may interconnect. FIG.2, illustrates that more than one route in the surgical data structure200 is possible. In certain embodiments, the preferred route 230 refersto a set of successful procedural routes which constitute a very smallsubsection of a much larger, multidimensional surgical data structure,where each node and edge are can include a description of and/orinformation corresponding to surgical states and actions respectively.In certain embodiments, there may be more than one successful proceduralroute. Graph theory concepts can then be applied to facilitate provisionof real-time guidance (e.g., identifying one or more edges to cause atransition between two nodes defined for a surgery) or to facilitatepost-hoc data analysis of one or more surgeries (e.g., to identify whichpaths are associated with particular consequences, an extent to which apredefined procedure was followed, etc.).

FIG. 3 illustrates an embodiment of the surgical data structure 300where surgical actions are associated with nodes 310 and proceduralstates are associated with edges 330. FIG. 3 uses the carpal tunnelprocedure to illustrate an alternate embodiment of a surgical map. FIG.3 illustrates the four key procedural states associated with edges 330,332, 334, and 336. Each key procedural state includes metadata toprovide a description of and/or information corresponding to theprocedural state.

In the embodiment illustrated by FIG. 3, the first node 310 isassociated with a procedural action. The action metadata 308 includesinstructions and tools necessary to perform the procedural action. Thefirst node 310 includes instructions to position the patient. The firstkey procedural state is associated with a first edge 330. The first edge330 is associated with a set of procedural metadata 312 and includes adescription of and/or information corresponding to the first keyprocedural state. The preferred route 340, shown by the dashed line inFIG. 3, connects each of the four key procedural states. A second edge332 is associated with a second set of procedural metadata 314indicating physiological states associated with the second keyprocedural state. A third edge 334 is associated with a third set ofprocedural metadata 316 indicating physiological states associated withthe third key procedural state. A fourth edge 336 is associated with afourth set of procedural metadata 318 indicating physiological statesassociated with the fourth and final key procedural state. A pluralityof nodes 322 are associated with procedural actions and action metadataconnect the key procedural states associated with edges 330, 332, 334,and 336

FIG. 4 illustrates a surgical data structure 400 that can recommend apreferred surgical route 424 based on weights assigned to one or moreedges between the node-to-node transitions. In some embodiments, aweight may be assigned to each node in addition to, or in lieu of, theweight assigned to each edge. The weights may be associated with aplurality of factors including, (for example) surgical outcomes, risk,prevalence of use, current procedural state, patient characteristics,vital signs, procedure specific parameters, and/or user (e.g., surgeon,nurse or team member) experience. The risk may be based on maximum riskor a risk based on the probability of occurrence. The weights may bemanually assigned when constructing the surgical data structure. Inother embodiments, the surgical weights may be determined or updatedusing a machine learning algorithm based on collected surgical data,including surgical outcomes. In some embodiments, a user (e.g., surgeon,nurse or team member) may update the weights pre-surgery using asurgical data structure explorer.

The weight 450 assigned to an edge 412 may be based on a number offactors. In some embodiments, the weight 450 may be associated with theprobability of a transition from a first node to a second node. In FIG.4, the weight 450 is assigned to a first edge 412 and a second weight452 is assigned to a second edge 414. In this example, the weight 450 islower than the second weight 452 and the lower weight may be associatedwith a preferred edge. The second weight 452 may be higher because inorder to arrive at a preferred node 420, a surgeon must perform twosurgical actions and pass through an intermediate node 416. In selectingthe second edge 414, the surgeon performs a surgical action associatedwith the second edge 414 to reach intermediate node 416. Next, thesurgeon must complete the surgical action associated with intermediateedge 418 to arrive at the preferred node 420.

In some embodiments the systems and methods described herein can use theweights assigned to the edges of the surgical data structure todetermine a preferred route 424, illustrated as a dashed line in FIG. 4.The preferred route 424 may be determined by adding the weights of theassociated edges to calculate a cost of traversing from an initial node410 in the surgical data structure to a destination node 440 in thesurgical data structure. In other embodiments, the preferred route maybe recalculated using a current node and a target node that are a subsetof nodes in the surgical data structure. The target node can be a nodein the surgical data structure selected by (for example) a surgeon, anoperating room team member, control room personnel, and/or one or moreautomated methods described herein.

The systems and methods described herein may calculate a cost for aroute in the surgical data structure. Determining the cost may depend onwhether weights are included in the surgical data structure. The costmay be calculated based on how values are assigned to the weights in thesurgical data structure. In some embodiments, the lowest cost route maybe the route selected. In other embodiments, the cost may be just oneinput into a processor that is configured to determine a route from aninitial node to a destination node.

In other embodiments, the weights associated with the edges of asurgical data structure may be associated with an actual value and amaximum value. Weight 454 illustrates that edge 426 may be selected ifactual value ‘a’ is less than the capacity ‘c’. In some embodiments, thecapacity may represent patient vital signs. Edge 428 may be associatedwith a surgical action that may be performed if a set of vital signs arebelow a set of threshold values. Vital sign weight 456 can representthat edge 428 is available for selection if less than or equal to ‘2’vital signs are below threshold values. For example, the ‘1’ in vitalsign weight 456 can indicate that the patient's heart rate is above athreshold value but blood pressure and breathing rate are at acceptablelevels. As a result, edge 428 is available for selection to transitionfrom a present node 422 to a destination node 440.

In some embodiments, the structures and technologies described hereininclude a global surgical graph 402 that may include a map to aplurality of pre-coded surgical data structures. During a surgicalprocedure using a fully developed pre-coded surgical data structure,there is a very low probability of a surgical complication that maycause a patient to transition to a procedural state that is notassociated with a node. For example, the first pre-coded surgical datastructure 404 may start at node 410 and represent the carpal tunnelprocedure discussed above. Some rare carpal tunnel complications thatmay still need to be accounted for include heart attacks and pulmonaryaspiration. During the carpal tunnel procedure, the patient maytransition to a procedural state associated with a complication that isnot represented by a node in the first pre-coded surgical data structure404.

To account for low probability complications, the systems and methodsdescribed herein may transition to another pre-coded surgical datastructure in the global surgical graph 402. For example, during thecarpal tunnel procedure, the global surgical graph 402 can include asecond pre-coded surgical data structure 406 associated with treating apatient experiencing a heart attack, and a third pre-coded surgical datastructure 408 associated with treating a patient experiencingaspiration. The second pre-coded surgical data structure 406 begins atnode 460, which includes a description of and/or informationcorresponding to a procedural state and patient attributes associatedwith a heart attack. The third pre-coded surgical data structure 408begins at node 470, which includes a description of and/or informationcorresponding to a procedural state and patient attributes associatedwith pulmonary aspiration. The second pre-coded surgical data structure406 and the third pre-coded surgical data structure 408 continue beyondline 462 to a node associated with a desired outcome for each procedure.

In some embodiments, the systems and methods described herein mayautomatically transition between the surgical data structures includedin the global surgical graph 402. In addition to the automatictransition, the systems and methods described herein may automaticallytransmit an alert to a specialist to assist with the associatedcomplication. In alternate embodiments, the control center may monitorsurgical data and determine that the current procedural state is notassociated with a node in the present surgical data structure. Thecontrol center may select an appropriate surgical data structure totransition to based, for example, on the procedural state, surgicaldata, and/or other factors related to the patient state. The controlcenter may page a specialist based on the severity of the complication.

In certain embodiments, the surgical data structure may be customized todirect surgery using the structures and technologies described herein.For example, a user (e.g., surgeon, nurse or team member) may access asurgical data structure prior to surgery and manually select anyacceptable procedural route before surgical navigation is initiated.Furthermore, the whole procedure may be specifically planned, such thata tailored route (where a user chooses every node and edge) can bedefined before the procedure. The structures and technologies describedherein may provide a user (e.g., surgeon, nurse or team member) thecapability to edit a surgical data structure using a softwareapplication such as a surgical data structure explorer. In alternateembodiments, the structures and technologies described herein may beconfigured to recommend the most appropriate surgical route, dependenton patient specific data (e.g., and/or live data). Further, the weightsassociated with the surgical data structure may be specifically set bythe systems and methods described herein or by a user (e.g., surgeon,nurse or team member) before a specific surgical procedure.

In some embodiments, machine learning may be used to identify an actionto recommend in a given circumstance (e.g., current state, patientcharacteristic, vital sign) and/or to a given operating room teammember. Action recommendation may be based on assessing potential and/orexpected risk associated with each of one or more actions. For example,a potential risk may be defined as a maximum risk impact or a weightedsum of risk impact (weighted based on probability of occurrence), and aselection may then be made to select an action with a lowest, low orbelow-threshold risk impact.

FIG. 5 provides a schematic illustration of communication exchangesbetween the structures and technologies described herein. The structuresand technologies described herein includes operating room interfacedevices 510, patient data devices 530, surgeon interface devices 540,local server 560, and cloud server 580. The operating room interfacedevices 510 can include operating room cameras and microphones,non-surgeon operating room team member headsets, smart surgical tools,power usage, light source usage, suction devices, irrigation devices.The patient data devices 530 can include any patient monitoringequipment including vital signs, patient discharge measuring devices,and ventilation volume measurements. Surgeon interface devices 540 caninclude an interface device, such as a headset and additional surgeonmonitoring devices. Additional surgeon monitoring devices may includegalvanic skin response devices and accelerometers to determine surgeonstress levels. As illustrated, the local server 560 may generate andtransmit a request for the current surgical data structure 562 from thecloud server 580. The request may include version information includinga version number, a date, or other indication of the local serverversion of the surgical data structure.

The cloud server 580 will perform a request evaluation 502 to determineif the local server 560 has the current version of the surgical datastructure. In performing the request evaluation 502, the cloud serverwill determine whether the local server 560 has the current surgicaldata structure. After evaluating the request, the cloud server 580 willgenerate and transmit a surgical data structure update 584 to the localserver 560. If the local server has the current version of the surgicaldata structure, the contents of the surgical data structure update 584will be limited. If the surgical data structure is not current, thesurgical data structure update 584 will include updates for theassociated surgical data structure. In some embodiments, the request forthe current surgical data structure 562 may be associated with asurgical data structure not present on the local server 560. In thiscase, an entire surgical data structure may be included in the surgicaldata structure update 584.

Upon receiving the surgical data structure update 584 the local server560 may generate and transmit a start procedure message 568 to thesurgeon interface devices, the patient data devices, and the operatingroom interface devices. The start procedure message 568 may includemachine readable instructions that instruct the devices to begincollecting surgical data associated with the surgical procedure. Eachdevice may generate and transmit surgical data to the local server.

Upon receipt of the start procedure message 568, the surgeon interfacedevices 540 and the operating room interface devices 510 will presentrelevant procedural information to the surgeon and the operating roompersonnel respectively. Additionally, upon receipt of the startprocedure message 568, the surgeon interface devices 540, the patientdata devices 530, and the operating room interface devices 510 willbegin recording and transmitting data to the local server 560. The datarecorded and transmitted by the surgeon interface devices 540, thepatient data devices 530, and the operating room interface devices 510may be categorized as surgical data 564.

Specific surgical data is generated by each of the devices in thestructures and technologies described herein. The surgeon interfacedevices 540 generate audio and/or visual data (A/V data) associated withthe surgeon's actions. The surgeon interface device 540 may beconfigured to receive inputs from a surgeon. Inputs may be auditory,visual, or haptic. In some embodiments, the surgeon's inputs may be inresponse to a stimulus. The surgeon A/V data 542 and the surgeon inputs544 are transmitted to the local server 560. The patient data devices530 generate and transmit vital sign data 532 and patient state data 534to the local server 560. Data associated with operating room interfacedevices 510 includes anesthetist data 512, operating room team memberinputs 516, operating room team member data 514, operating room visualdata 520, and operating room surgical instrument data 518. The dataassociated with operating room interface devices 510 is also transmittedto local hospital server 560.

Next, the local server 560 performs feature extraction 504 anddetermines a current node 506. Feature extraction 504 is performed bythe local server 560 using the surgical data 564 generated andtransmitted by the operating room interface devices 510, the patientdata devices 530, and the surgeon interface devices 540. The localserver 560 may detect components related to the procedural metadatastored in the surgical data structure using feature extraction 504. Forexample, components may include anatomical states, phonemes, and/orsurgical tools. In some embodiments, the feature extraction 504 caninclude extracting visual or auditory components from the live surgicaldata.

Local server 560 determines a current node 506 using the surgical datastructure and the components extracted from the surgical data 564. In analternate embodiment, the local server 560 forwards surgical data 564 bto the cloud server 580. The cloud server may perform feature extraction504 b, determine a current node 506 b, and transmit the current node 596to local server 560.

Based on the current node the local server 560 generates and transmitssurgeon guidance 566, and OR team member guidance 564. The guidance mayinclude information from the metadata associated with the current nodeand/or information related to a procedural action associated with anedge to transition from the current node to a target node. In someembodiments, the surgical data structure (or a version thereof) may beupdated (e.g., at the cloud server 580 or at the local server 560)during a surgical procedure or at another time. For example, the updatemay be based on the surgical data 564 b received at the cloud server580. The cloud server 580 may generate and transmit a surgical datastructure update 588 to the local server 560 during the surgicalprocedure. The Operating room interface devices 510, patient monitoringdevices 530, and surgeon interface devices 540 continue generating andtransmitting surgical data, and the local server 560 continuesextracting features and determining current nodes until the local server560 determines the surgical procedure is at a final node 508. In someimplementations, the local server 560 may transmit surgical data 564 bto the cloud server 580, which determines the surgical procedure is at afinal node 508 b.

Upon determining the surgical procedure is complete, the local server560 may transmit a procedure complete message 570 to the operating roominterface devices 510, the patient data devices 530, and the surgeoninterface devices 540. In response, the devices may present informationto operating room team members and the surgeon that indicate thesurgical procedure is complete. Further, the devices may stoptransmitting surgical data to the local server upon receiving theprocedure complete message 570. In some implementations, the localserver 560 may transmit a complete record of the surgical data 572associated with the surgical procedure to the cloud server for analysis.

After receiving the complete record of surgical data 572, the cloudserver may analyze collected surgical data 590. Analyzing the collectedsurgical data may include determining updates to the surgical datastructure, determining user (e.g., surgeon, nurse or team member)compliance with procedure guidelines, and annotating the record ofsurgical data to indicate data associated with procedural states.Updates to the surgical data structure may include (for example)updating the metadata associated with current nodes and edges,generating new nodes, generating new edges to transition between currentnodes and new nodes, updating the weights associated with edges, and/ordeleting nodes/edges no longer present in a surgical procedure.

In some embodiments, analyzing the collected surgical data may includegenerating an annotated video file that includes video data andannotations associated with the nodes and edges present in the videodata. In other embodiments, analyzing the collected surgical data mayinclude generating compliance data that identifies entities involved inthe surgical procedure and any departures from the preferred surgicalguidelines. The compliance data may be stored in a compliance dataset.The compliance dataset can be transmitted to (for example) the surgeonsthat performed the procedure, medical schools, and/or medical reviewboards.

In some embodiments, analyzing the collected surgical data may includeintegrating the present set of surgical data with a global set ofsurgical data. The remote server can be configured to process the globalset of surgical data and apply machine learning techniques to update thesurgical data structure based on the global surgical data. For example,the global surgical dataset may contain distinct features and componentsthat are not linked with a node in the associated surgical datastructure. In response, the remote server may define a new discreteprocedural state and update the associated surgical data structure withthe appropriate nodes and edges.

FIGS. 6A, 6B, and 6C illustrate embodiments of heads-up displays used topresent surgical guidance to operating room personnel during a surgicalprocedure. The information presented may be included in metadataassociated with a current node, a target node or an edge in the currentsurgical data structure. FIGS. 6A, 6B, and 6C illustrate views of theoperating room from a surgeon or operating room personnel-mountedheadset. In addition to presenting information, the headset may beequipped to collect live surgical data. For example, data may becollected via a camera, speaker, haptic sensor, and/or receiver (e.g.,receiving data from a smart surgical tool). Data may be collectedcontinuously or periodically (e.g., at defined times, at defined timeintervals or upon detecting an event triggering collection).

In some embodiments, a processing device, such as the local server, maydetermine and continuously update a trajectory through the surgical datastructure. The trajectory may be based on (for example) live surgicaldata, weights associated with nodes and edges the surgical datastructure, changes in patient/surgery states, and/or a pre-programmedtrajectory entered by a user. A user may enter a pre-programmedtrajectory using a software application, such as a surgical map. Thelocal server may transmit the trajectory data to (for example) one ofthe heads up displays and/or the control center to increase situationalawareness.

FIG. 6A illustrates an augmented reality view 600 that can be used topresent the steps associated with a surgical procedure from a surgeon'spoint-of-view (POV) heads-up display. The display can be semitransparentpermitting the surgeon to see the patient 690 and surrounding operatingroom through the heads-up display. The graphics overlaid on thesurgeon's POV include a primary display area 610, a first supportingdisplay area 620, a second supporting display area 630, a title area650, a procedural progress timeline 652 and indicator 654, and anelapsed time indicator 656. The display includes data retrieved from asurgical data structure. The local server can determine the relevantdata based on live surgical data collected during a procedure.

The primary display area 610 may be used to present useful proceduralinformation into the surgeon's field of view. Information in the primarydisplay area may include (for example) identification of one or moreactions, tools to be used, and/or expected results of an action. In someinstances, data is presented in the primary display area 610, the firstsupporting display area 620, and/or the second supporting display area630 only upon detecting a presentation-triggering event (e.g., that anorientation of a wearable device matches an orientation profile, that anacceleration of a wearable device matched an acceleration profile, orthat a physical input of a predefined type was received).

In FIG. 6A, the primary display area includes a list of key stepsassociated with distal femoral preparation. The title 612 indicates thatkey steps are displayed. The steps are provided in a primary displaylist 614 with the current step highlighted 616. The first supportingdisplay 620 includes a list of phase objectives. First supportingdisplay title 622 indicates that content provided in first supportingdisplay 620. The first supporting display list 624 is displayed toprovide the surgeon with detailed phase objectives. The secondsupporting display 630 includes a video 632 of the current key step tobe performed. The surgeon, operating room personnel, and/or the controlcenter may remove any of the information from the field of view. Thelocal server may automatically remove items from the surgeon's field ofview based on surgical data received by the system.

FIG. 6B illustrates an augmented reality view 600 to present informationassociated with a surgical procedure to a non-surgeon operating roomteam member. The display can be semitransparent permitting the teammember to see an operating room table 692, operating room instruments626, and the surrounding operating room through the heads-up display.The graphics overlaid on the team member's POV include a primary displayarea 610, a first supporting display area 620, a second supportingdisplay area 630, a title area 650, a procedural progress timeline 652and indicator 654, and an elapsed time indicator 656.

The primary display area 610 may be used to present useful proceduralinformation into the operating room team member's field of view. In FIG.6B, the relevant operating room team member is a nurse. For a nurse,useful procedural information may include the required instruments forthe current phase of a procedure. The title 612 on the nurse's primarydisplay 610 reflects the difference in relevant information. the primarydisplay list 614 lists the required instruments and the currentinstrument is highlighted 616. The augmented reality view 600 alsoincludes an instrument indicator 618. The local server may use featureextraction from the associated headset's live video data to identify andhighlight relevant instruments using an instrument indicator 618. Theinstrument 628 associated with the instrument indicator 618 maycorrespond to the highlighted 616 instrument on the primary display 610.

FIG. 6C illustrates an augmented reality view 600 used to capture asnapshot during the surgical procedure. A snapshot is useful forpost-surgery analysis of the surgical procedure. The snapshot viewincludes a title area 650, a procedural progress timeline 652 andindicator 654, an elapsed time indicator 656, a set of framing elements680, and a visible timestamp 682.

In addition to the visual data captured by the snapshot in FIG. 6C, thelocal server may record additional data associated with the snapshot.Additional sensors that can provide additional data associated with thesnapshot can include at least one operating room video camera, amicrophone, an infrared sensor and an accelerometer. The additionalsensors can be used to capture real-time data from the operating roomincluding video, pictures, audio annotation, manual procedural steplogging and user (e.g., surgeon, nurse or team member) notes. Forexample, the user may use the head mounted camera to capture imagestills of critical steps of surgery for documentation of their practicein the event of an unexpected complication.

In some embodiments the captured images, and any other data associatedwith additional sensors captured during surgery can be stored againstthe active surgical procedure at the precise procedural location atwhich it was measured. This data may be used as, for example, an auditrecord, or training resource. Further, these sensors may facilitateuser-system interactions, which may include gestures, voice commands,other form of hands-free interaction or direct hardware interactionssuch as buttons, or pedals, which may be separate to the headset butconnected to structures and technologies described herein to providereal-time surgical guidance.

The guidance presented to the surgeon and operating room personnel isnot limited to the embodiments shown herein. Surgical guidance may takethe form of text commands, images (separate and/or overlaid on real-timevideo), augmented reality (AR) or virtual reality (VR) surgical guides(such as paths of incisions), informative animations (both AR, VR andseparate from real image of surgery), haptic feedback and audio orvisual commands or stimulus (e.g., signals, alarms, music, and ambientlight change).

With reference to now to FIG. 7, a block diagram of an illustrativesurgical navigation local server 700 is shown. The local server 700 maycorrespond to the local server 170 described above. Local server 700includes processing units 704 that communicate with a number ofperipheral subsystems via a bus subsystem 702. These peripheralsubsystems include, for example, a storage subsystem 710, an I/Osubsystem 726, and a communications subsystem 732.

Bus subsystem 702 provides a mechanism for letting the variouscomponents and subsystems of local server 700 communicate with eachother. Although bus subsystem 702 is shown schematically as a singlebus, alternative embodiments of the bus subsystem may utilize multiplebuses. Bus subsystem 702 may be of any several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of bus architectures. Sucharchitectures may include, for example, an Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnect (PCI) bus, which may beimplemented as a Mezzanine bus manufactured to the IEEE P1386.1standard.

Processing unit 704, which may be implemented as one or more integratedcircuits (e.g., a microprocessor or microcontroller), controls theoperation of local server 700. Processing unit 704 may be implemented asa special purpose processor, such an application-specific integratedcircuit (ASIC), which may be customized for a particular use and notusable for general-purpose use. In some implementations, an ASIC may beused to increase the speed of feature extraction from live surgicaldata. In some embodiments, processing unit 704 may include one or moregraphics processing units (GPUs). The GPUs may be configured to processlive surgical data in order to perform feature extraction. One or moreprocessors, including single core and/or multicore processors, may beincluded in processing unit 704. As shown in FIG. 7, processing unit 704may be implemented as one or more independent processing units 706and/or 708 with single or multicore processors and processor cachesincluded in each processing unit. In other embodiments, processing unit704 may also be implemented as a quad-core processing unit or largermulti core designs (e.g., hexacore processors, octo-core processors,ten-core processors, or greater). In some embodiments, one or moreindependent processing units may be replaced with an ASIC optimized forfeature extraction.

Processing unit 704 may execute a variety of software processes embodiedin program code, and may maintain multiple concurrently executingprograms or processes. At any given time, some or all of the programcode to be executed may be resident in processor(s) 704 and/or instorage subsystem 710. In some embodiments, local server 700 may includeone or more specialized processors, such as digital signal processors(DSPs), outboard processors, graphics processors, application-specificprocessors, and/or similar specialized processors.

I/O subsystem 726 may include device controllers 728 for one or moreuser interface input devices and/or user interface output devices. I/Odevices can include a user (e.g., surgeon, nurse or team member) I/Odevice 730 for use by a user pre, during or post-surgery. A surgicaldata structure explorer can be executed on the user I/O device 730. Theuser I/O device 730 can provide a user the capability to adjust thepreferred route, procedural metadata, assigned weights, and/or otherdata associated with a surgical data structure prior to surgery.Post-surgery a user may use the user I/O device 730 to review theanalysis performed by a processing device. In some embodiments, the I/Odevices can include control center I/O devices 740.

User interface input and output devices may be integral with localserver 700 (e.g., integrated audio/video systems, and/or touchscreendisplays), or may be separate peripheral devices which areattachable/detachable from local server 700. The I/O subsystem 726 mayprovide one or several outputs to a user by converting one or severalelectrical signals to user perceptible and/or interpretable form, andmay receive one or several inputs from the user by generating one orseveral electrical signals based on one or several user-causedinteractions with the I/O subsystem such as the depressing of a key orbutton, the moving of a mouse, the interaction with a touchscreen ortrackpad, the interaction of a sound wave with a microphone, or similaruser inputs.

Input devices may include a keyboard, pointing devices such as a mouseor trackball, a touchpad or touch screen incorporated into a display, ascroll wheel, a click wheel, a dial, a button, a switch, a keypad, audioinput devices with voice command recognition systems, microphones, andother types of input devices. Input devices 730 may also include threedimensional (3D) mice, joysticks or pointing sticks, gamepads andgraphic tablets, and audio/visual devices such as speakers, digitalcameras, digital camcorders, portable media players, webcams, imagescanners, fingerprint scanners, barcode reader 3D scanners, 3D printers,laser rangefinders, haptic devices, and eye gaze tracking devices.Additional input devices may include, for example, motion sensing and/orgesture recognition devices that enable users to control and interactwith an input device through a natural user interface using gestures andspoken commands, eye gesture recognition devices that detect eyeactivity from users and transform the eye gestures as input into aninput device, voice recognition sensing devices that enable users tointeract with voice recognition systems through voice commands, medicalimaging input devices, MIDI keyboards, digital musical instruments,electromagnetic sensors configured to detect movement of surgical tools,optical sensors, near infrared sensors, and similar devices.

Output devices may include one or more display subsystems, indicatorlights, or non-visual displays such as audio output devices, etc.Display subsystems may include, for example, cathode ray tube (CRT)displays, flat-panel devices, such as those using a liquid crystaldisplay (LCD) or plasma display, light-emitting diode (LED) displays,projection devices, touch screens, haptic devices, and the like. Ingeneral, use of the term “output device” is intended to include allpossible types of devices and mechanisms for outputting information fromlocal server 700 to a user or other computer. For example, outputdevices may include, without limitation, a variety of display devicesthat visually convey text, graphics and audio/video information such asmonitors, printers, speakers, headphones, automotive navigation systems,plotters, voice output devices, and modems.

Local server 700 may comprise one or more storage subsystems 710,comprising hardware and software components used for storing data andprogram instructions, such as system memory 718 and computer-readablestorage media 716. The system memory 718 and/or computer readablestorage media 716 may store program instructions that are loadable andexecutable on processing units 704, as well as data generated during theexecution of these programs. Program instructions may includeinstructions to perform one or more actions or part(s) or all of one ormore methods or processes described herein. For example, programinstructions may include instructions for identifying a discretesurgical procedural state. Program instructions may include instructionsfor generating, transmitting, and/or receiving communications. Programinstructions may include instructions for automated processing. Programinstructions may include instructions for generating automatedprocessing and/or stage results. Program instructions may includeinstructions for performing a work flow iteration.

Depending on the configuration and type of local server 700, systemmemory 718 may be stored in volatile memory (such as random accessmemory (RAM) 712) and/or in non-volatile storage drives 714 (such asread-only memory (ROM), flash memory, etc.) The RAM 712 may include dataand/or program modules that are immediately accessible to and/orpresently being operated and executed by processing units 704. In someimplementations, system memory 718 may include multiple different typesof memory, such as static random access memory (SRAM) or dynamic randomaccess memory (DRAM). In some implementations, a basic input/outputsystem (BIOS), including the basic routines that help to transferinformation between elements within local server 700, such as duringstart-up, may typically be stored in the non-volatile storage drives714. By way of example, and not limitation, system memory 718 mayinclude surgical data structure storage 720, recorded surgical data 722,and application programs 724 such as feature extraction applications,procedural state identification applications, headset displayapplications, surgical data structure explorer applications, userapplications, mid-tier applications, server applications, etc.

Storage subsystem 710 also may provide one or more tangiblecomputer-readable storage media 716 for storing the basic programmingand data constructs that provide the functionality of some embodiments.Software (programs, code modules, instructions) that when executed by aprocessor provide the functionality described herein may be stored instorage subsystem 710. These software modules or instructions may beexecuted by processing units 704. Storage subsystem 710 may also providea repository for storing data used in accordance with the embodimentsillustrated herein.

Storage subsystem 710 may also include a computer-readable storage mediareader that may further be connected to computer-readable storage media716. Together and, optionally, in combination with system memory 718,computer-readable storage media 716 may comprehensively representremote, local, fixed, and/or removable storage devices plus storagemedia for temporarily and/or more permanently containing, storing,transmitting, and retrieving computer-readable information.

Computer-readable storage media 716 containing surgical data structures,program code, or portions of program code, may include any appropriatemedia known or used in the art, including storage media andcommunication media, such as but not limited to, volatile andnon-volatile, removable and nonremovable media implemented in any methodor technology for storage and/or transmission of information. This mayinclude tangible computer-readable storage media such as RAM, ROM,electronically erasable programmable ROM (EEPROM), flash memory or othermemory technology, CD-ROM, digital versatile disk (DVD), or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or other tangible computerreadable media. This may also include nontangible computer-readablemedia, such as data signals, data transmissions, or any other mediumthat may be used to transmit the desired information and that may beaccessed by local server 700.

By way of example, computer-readable storage media 716 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray disk, or other optical media. Computer-readablestorage media 716 may include, but is not limited to, Zip drives, flashmemory cards, universal serial bus (USB) flash drives, secure digital(SD) cards, DVD disks, digital video tape, and the like.Computer-readable storage media 716 may also include, solid-state drives(SSD) based on non-volatile memory such as flash-memory based SSDs,enterprise flash drives, solid state ROM, and the like, SSDs based onvolatile memory such as solid state RAM, dynamic RAM, static RAM,DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs thatuse a combination of DRAM and flash memory based SSDs. The disk drivesand their associated computer-readable media may provide non-volatilestorage of computer-readable instructions, data structures, programmodules, and other data for local server 700.

Communications subsystem 732 may provide a communication interface fromlocal server 700 and remote computing devices via one or morecommunication networks, including local area networks (LANs), wide areanetworks (WANs) (e.g., the Internet), and various wirelesstelecommunications networks. As illustrated in FIG. 7, thecommunications subsystem 732 may include, a remote server networkinterface controller (NIC) 734. The NIC may be an Ethernet card,Asynchronous Transfer Mode NICs, Token Ring NICs or a similar interface.The communications subsystem 732 may include one or more wirelessinterfaces, including an operating room communications interface 738.The operating room communications interface 738 may be implemented usinga wireless network interface controller. Additionally and/oralternatively, the communications subsystem 732 may include one or moremodems (telephone, satellite, cable, ISDN), synchronous or asynchronousdigital subscriber line (DSL) units, FireWire interfaces, USBinterfaces, and the like. Communications subsystem 732 also may includeradio frequency (RF) transceiver components for accessing wireless voiceand/or data networks (e.g., using cellular telephone technology,advanced data network technology, such as 3G, 4G or EDGE (enhanced datarates for global evolution), Wi-Fi (IEEE 802.11 family standards, orother mobile communication technologies, or any combination thereof),global positioning system (GPS) receiver components, and/or othercomponents. A cellular enabled local server 700 may be used in austereconditions where users (e.g., surgeon, nurse or team member) maytraditionally have limited access to traditional methods of surgicalguidance.

The various physical components of the communications subsystem 732 maybe detachable components coupled to the local server 700 via a computernetwork, a Fire Wire bus, a serial bus, or the like, and/or may bephysically integrated onto a motherboard or circuit board of localserver 700. Communications subsystem 732 also may be implemented inwhole or in part by software.

In some embodiments, communications subsystem 732 may also receive inputcommunication in the form of structured and/or unstructured data feeds,event streams, event updates, and the like, on behalf of one or moredevices that may use or access local server 700. For example,communications subsystem 732 may be configured to receive data feedsassociated with surgical data or surgical data structures in real-timefrom other communication services, web feeds such as Rich Site Summary(RSS) feeds, and/or real-time updates from one or more third partyinformation sources. Additionally, communications subsystem 732 may beconfigured to receive data in the form of continuous data streams, whichmay include event streams of real-time events and/or event updates(e.g., surgical data, surgical data structure updates, preferred routeupdates, patient updates, etc.). Communications subsystem 732 may outputsuch structured and/or unstructured data feeds, event streams, eventupdates, and the like to one or more data stores that may be incommunication with device 700.

Due to the ever-changing nature of computers and networks, thedescription of local server 700 depicted in FIG. 7 is intended only as aspecific example. Many other configurations having more or fewercomponents than the device depicted in the figure are possible. Forexample, customized hardware may also be used and/or particular elementsmay be implemented in hardware, firmware, software, or a combination.Further, connection to other computing devices, such as networkinput/output devices, may be employed. Based on the disclosure andteachings provided herein, it will be appreciated that there are otherways and/or methods to implement the various embodiments.

Referring next to FIG. 8, a flowchart of a process 800 for the automatedprovisioning of real-time custom procedural surgical guidance 800 isillustrated. The method for the automated provisioning of real timecustom procedural guidance includes a plurality of steps connected withdirectional arrows. The directional arrows not only indicate that theelements are connected, but also indicate the direction that data mayflow with respect to the various elements. For example, first, themethod receives a surgical data structure 802. For example, the surgicaldata structure may be received from another remote device (e.g., cloudserver), stored and then retrieved prior to or during performance of asurgical procedure. The surgical data structure can provide precise andaccurate coded descriptions of complete surgical procedures. Thesurgical data structure 802 can describe routes through surgicalprocedures and is described in detail above with reference to FIG. 2.For example, the surgical data structure 802 can include multiple nodes,each node representing a procedural state (e.g., corresponding to astate of a patient). The surgical data structure 802 can further includemultiple edges, with each edge connecting multiple nodes andrepresenting a surgical action.

After receiving the surgical data structure, live surgical data isreceived at block 804. In certain embodiments, live surgical data 804may include user inputs, a video stream, or operating room data. As oneillustration, user inputs may include manual inputs directly provided byoperating room personnel (e.g., via a wearable device or non-wearabledevice located inside or near an operating room). For example, operatingroom personnel may include the performing surgeon, a nurse, ananesthetist, or remote surgeon watching the procedure. Inputs mayinclude gesture, voice, another form of hands-free sterile interactionand even direct contact with a computer user-interface. As another(additional or alternative) illustration, live surgical images may berecorded, such as those captured on video from a head-mounted surgeoncamera. As yet another (additional or alternative) illustration, livesurgical data 804 may include other operating room data such as; timeelapsed in procedure, nodes already passed by the users (e.g., surgeon,nurse or team member), identified surgical instruments used or in use,personnel present in the operating room and user movements measured fromaccelerometers in a guidance system headset or other system-connectedwearable devices.

At block 806, a component is detected from the live surgical data. Insome embodiments, the method detects a component from the live surgicaldata 806 using feature extraction. Anatomic feature extraction can beperformed on the live surgical images to detect an anatomic state,anatomic pose, or any other attribute possible to derive from video thatcan be used later to determine a precise procedural state. Additionaldata such as user input or operating room data may assist with featureextraction. The feature extraction algorithm used to detect a componentfrom the live surgical data may include machine learning iterations. Inparticular, if the first implementation of the systems and methodsdescribed herein requires manual navigation through the procedure by theuser (e.g., surgeon, nurse or team member), the resulting dataset oflive surgical images and associated navigation commands can be used asan effective algorithm training resource using machine learningsoftware. In other embodiments, a component may be associated with auser input or operating room data.

At block 808, a current node from the surgical data structure isidentified. In some embodiments, the current node will correspond to thecomponent detected from the live surgical data. For example, a componentmay identify a patient state, which may indicate which of multiple nodescorresponds to a current circumstance. In other embodiments, an edge maycorrespond to the component detected in the live surgical data. Thecurrent node and the component may correspond to a discrete proceduralstate. In other embodiments, the user input or the operating room datamay be used to determine the current node from the surgical datastructure.

At block 810, procedural information for the current node is retrieved.For example, each of one, more or all nodes may have particularprocedural information associated with the node, and some or all of theparticular procedural information may be retrieved. As another example,a current node may be connected to each of one or more next-stage nodesvia a corresponding edge. An edge may have particular proceduralinformation associated with the edge, and some or all of the particularprocedural information may be retrieved. Procedural information caninclude a description of and/or information corresponding to a patientstate that fully describes the condition of a patient at a given pointduring a medical procedure. Procedural information may include (forexample) pose, anatomical state, initial preparation and/orsterilization information. In some embodiments, the quantity ofprocedural information to be provided may be tailored e.g., to loweroperating room personnel information workload.

At block 812, the procedural information is transmitted. In someembodiments, the method may transmit the procedural information to awearable device used by one or more operating room personnel. In someembodiments, the process 800 can return from block 814 to receiving livesurgical data at block 804 and then repeat the blocks to detectcomponents 806, determine a new current node 808, retrieve proceduralinformation for the current node 810, and again transmit the proceduralinformation 812 for the current node. In some embodiments, thestructures and technologies described herein may transmit the proceduralinformation 812 based on a contingent event such as low point inoperating room activity or an input form the surgeon or the operatingroom personnel.

The structures and technologies described herein may provide real timerecommendations of surgical actions based on live data as described inprocess 900, as depicted in FIG. 9. At block 902, a procedural state isidentified. For example, a node may be identified from amongst multiplenodes represented in a surgical data structure. The identified node maycorrespond to a discrete procedural state corresponding to a currentcircumstance (and/or recently received live data) of a surgicalprocedure.

At block 906, one or more potential target nodes are identified. Eachpotential node can be connected (e.g., directly connected) to thecurrent node by an edge, which can be identified at block 908. Each ofthe one or more potential target nodes may include a node connected(e.g., directly connected) to the current node. Each of the one or morepotential target nodes and/or each node connecting the current node to apotential node of the one or more potential target nodes may beassociated with one or more variables, such as a weight, representing,for example, a potential surgical outcome, surgical risk, outcomeprobability and/or predicted time commitment. Surgical risk mayrepresent the predicted probability of an unfavorable outcome occurringat any point during surgery. Surgical risk may represent the probabilityof complications in a node-to-node transition.

The variable(s) may have been generated based on (for example) anoutcome of one or more real surgical procedures recorded by the guidancesystem, knowledge of outcome from annotated libraries of surgical videosand/or even appropriately tagged academic research relating to specificsurgeries or surgical steps. For example, a plurality of intra-surgicalindicators may be used to assess surgical risk and/or potential surgicaloutcomes. Direct surgical indicators may include, among other things,patient blood loss. The structures and technologies described herein maydetermine a patient who has undergone significant loss of blood is atmore risk than one who has reached the same point in the procedure withless blood lost. Indirect surgical risk indicators may include, amongother things, which instrument is being used, whether surgery iscurrently active near delicate anatomical structures and even simply thetime spent by the surgeon on each individual surgical step.

Further, surgical risk may also be stratified by patient specific data.Patient information such as age, weight, prior surgical interventionsand health history may further inform the calculation of surgical risk.Using collected metadata and other sources of risk indicators, such asacademic research, associations and relevant ‘weightings’ of each of atleast one of the one or more variables may be determined between patientspecific data (e.g., age) and live surgical parameters at specificprocedural-steps (e.g., blood loss during a specific phase of liversurgery).

The weighting of patient specific parameters may then be used tointerpret live surgical data to perform a selection of a target node atblock 910. The target node need not be a final node of the datastructure may correspond to a next (e.g., directly connected node). Forexample, a selection may be made to maximize (an outcome variable) orminimize a surgical risk or to maximize a score calculated based on acombination of multiple of the one or more variables.

At block 914, procedural information associated with the target node 912can be retrieved and transmitted (e.g., to a device present in or nearan operating room or location associated with the surgical procedure).

FIG. 10 shows a process 1000 for using live surgical data for trackingand procedural models. At block 1002, surgical data is received. Thesurgical data can include data collected during a performance of asurgery. The surgical data can include digital video data, which mayinclude a continuous signal that represents moving visual images. Thevideo data may, but need not, include an audio signal that is alignedwith the visual signal. The surgical data may (e.g., alternatively oradditionally) include audio data, representations of one or more inputsreceived at a device (e.g., a voice command, a motion command or inputreceived at a keyboard or other interface component), data from a smartsurgical tool (e.g., indicative of a location, movement and/or usage),and so on. The surgical data can include data collected from a singledevice or from multiple devices. The surgical data can include datacollected over a time period (e.g., a predefined time increment, since aprevious state-transition time or procedure initiation time, or a timeof an entire surgical procedure). The surgical data may include raw datacollected during the surgery or a processed version thereof (e.g., toreduce noise and/or transform a signal, such as transformingaccelerometer data collected at a headset to a movement commandindicating that the surgical procedure has progressed to a next step).

At block 1004, a procedural state is determined using all or part of thesurgical data. In some instances, the surgical data is incrementallyreceived (e.g., as it is collected in real-time), and surgical data canthen be incrementally processed (e.g., by separately processing eachdata increment or by aggregating a recently received data increment withother data, such as data from a previous time increment or from anotherdevice, and processing the aggregated data). In these instances, a statemay be determined for the new data (or aggregated data).

In some instances, the surgical data may include a larger data set thatmay represent multiple states. For example, a larger data set mayinclude a continuous data set (e.g., video collected over the course ofa surgical procedure). The surgical data may then be divided intomultiple chunks (e.g., of a predefined duration), and a state may bedetermined for each chunk. Thus, for example, block 1004 may beperformed multiple times.

The chunks may, or may not, be overlapping. When chunks are overlapping,an offset time by which the chunks advance may also be predefined. Forexample, in a 2-hour procedure, a video stream may be segmented into 60two-minute, non-overlapping segments. As another example, in a 2-hourprocedure, a video stream may be segmented to procedure 119 two-minutesegments, where each segment overlaps with a previous segment by 1minute. In some instances, surgical data includes a discrete data set.Discrete data points may be individually analyzed or binned into one ofmultiple time windows (e.g., which may, or may not be overlapping andmay have a predefined duration), such that each discrete data point maybe assessed in combination with other discrete data points and/or achunk of each of one or more continuous data sets that correspond to asame time window. In some instances, surgical data includesdeterministic indicators of state transitions, such as a commandreceived from a user indicating transition to a new state. Time windowsmay then be defined as times between these transitions, and othersurgical data may be binned in accordance with times associated with thedata.

Determining the procedural state may include, for example, selecting asingle node (or one or more nodes) from amongst a plurality of nodes ina surgical data structure. The surgical data structure can include oneassociated with a particular type of surgery being conducted (e.g., andmay be identified based on input received before, during or after thesurgery and/or based on a patient record or operating room schedule thatidentifies a type of surgical procedure being performed). The state maybe identified as one that correspond with the selected node(s). In someinstances, each node of a structure's plurality of nodes include one ormore live-data signatures that include one or more characteristics oflive data that are associated with the node. A characteristic mayinclude (for example) a value identifier (e.g., identifying a particularword command), a definition of a range (e.g., for a distance expectedbetween two particular devices), a threshold (e.g., a lower thresholdfor an acceleration value from a surgical tool), and so on. Thedetermination at block 1004 may include determining to which node'ssignature the received data most closely corresponds (and/or correspondswith to at least a threshold degree). In some instances, thedetermination can be constrained, such that only a subset of theplurality of nodes are considered (e.g., those nodes of the plurality ofnodes that are directly connected to a node associated with a currentstate). For example, the constraint may be such to not allow adetermination that a “Surgery-Complete” state would immediately follow a“Surgery-Begins” state.

Various transformation, detection and/or extraction techniques may beused to identify to which node the surgical data corresponds. Forexample, a model generated based on performance of a machine-learningalgorithm may be used to process the video data to identify the one ormore procedural states. As another example, audio cues from the videomay be assessed to identify the state(s). As yet another example,detecting that a particular tool is being used (e.g., based on amovement signal or activation signal received from the tool or based ondetecting that the tool is being moved to a location overlapping with apatient location based on video data) may be used to identify thestate(s).

It will be appreciated that one or both of blocks 1002 and 1004 may beperformed during a surgical procedure (e.g., at a local surgery in anoperating room or facility at which the procedure is being performed orat a remote server, such as a cloud server) or after the procedure usingstored data collected during the procedure. For example, in oneinstances, surgical data is collected during a surgical procedure bydevices located in an operating room, and subsequently a remote servermay receive and process the surgical data. As another example, a localdevice may receive and process the surgery data during the procedure.

In some embodiments, block 1004 may determine that the surgical datastructure does not include a procedural state that is associated withthe surgical data received at block 1002. Block 1004 may identify thatthe surgical data is associated with a new procedural state. In someembodiments, block 1004 may generate a new node using the processdescribed in FIG. 11. An update to the nodes and procedural statesidentified in the surgical data structure may be initiated automaticallyby a local or remote server or by a user input. The user input may beentered by, for example, a user in the control center with access to alocal server configured to execute surgical data structure explorer.

At block 1006, temporal information is identified for a part of thevideo data associated with the procedural state. The temporalinformation may include (for example) a start time, end time, durationand/or range. The temporal information may be absolute (e.g., specifyingone or more absolute times) or relative (e.g., specifying a time from abeginning of a surgery, from a beginning of a procedure initiation,etc.). The temporal information may include information defining a timeperiod or time corresponding to the video data during which it isestimated that the surgery was in the procedural state.

The temporal information may be determined (for example) based oninformation associated with the state identification. For example, ifsurgery data is divided into data chunks, and each data chunk isassociated with a procedural state, the temporal information mayidentify a start and end time that corresponds to (for example) a starttime of a first data chunk associated with a given procedural state andan end time of a last data chunk associated with a given proceduralstate, respectively. As another example, if discrete data points arereceived that are indicative of state transitions, the temporalinformation may include times associated with two consecutive datapoints. As yet another example, a data set may be analyzed to detectstate transitions (e.g., by detecting activation/inactivation of aparticular smart tool, detecting a time period during which a particularvisual characteristic is detected in video data, detecting times atwhich an audio command is detected in an audio signal that indicatesthat a state is transitioning, etc.). It will be appreciated thattransitions between different states may have a same or differenttransition characteristic.

At block 1008, electronic data is generated representing the part of thevideo data for the procedural state. The electronic data may includeinformation included in or identified by (e.g., via a locationidentification) metadata of a node associated with the procedural statein a surgical data structure. The electronic data may (for example)identify the procedural state (e.g., by identifying a state of apatient, progress of a surgery, etc.), identify an action that is beingperformed or is about to be performed (e.g., as identified in an edgethat connects a node corresponding to the procedural state with a nodecorresponding to a next procedural state), and/or identify one or moreconsiderations (e.g., a risk, tools being used or that are about to beused, a warning, etc.). The electronic data may relate to a trajectory.For example, electronic data may indicate when an action is beingperformed that is contrary to (or, in other instances, in accordancewith) a typical action, recommended action, or action prescribed by aguideline at a state of surgery.

The electronic data may include (for example) data that is to beconcurrently presented during presentation of at least part of the partof the video data. For example, the electronic data may include text(e.g., an annotation) that is to be overlaid or presented next to avisual display of the part of the video data. As another example, theelectronic data may include an audio signal (e.g., of spoken words) thatis to be presented during a visual display of the part of the videodata. Any audio data in the video signal may be, for example,eliminated, softened or maintained.

At block 1010, it is determined whether electronic data has beengenerated for all procedural states. If electronic data does not existfor all the procedural states, process 1000 returns to block 1004 or1006, where a next procedural state is determined or where temporalinformation is identified for another part of the video data associatedwith a next procedural state.

If electronic data exists for all procedural states, process 1000continues to block 1012, where the electronic data (e.g., generated foreach procedural state) is output. In some instances, the electronic datais aggregated across states. For example, modified video data may begenerated that includes the electronic data generated for eachprocedural state. The modified video data may include all or part oforiginal video data but with additional presentations of electronic datathat represents a procedural state represented in a displayed part ofthe video signal. For example, if a one-hour surgery included sixprocedural states—each associated with successive, non-overlapping10-minute time windows, modified surgery data may include a 60-minutevisual signal accompanied by an annotation that changes every 10 minutesor that appears (with new text every ten minutes). As another example,for the same hypothetical surgery, an annotation may repeatedly change(e.g., every minute) during the video, but the text may always be textassociated with a procedural state associated with a current image beingdisplayed. As yet another example, a video signal may be parsed toidentify a predefined number of images (e.g., 1 or a number greaterthan 1) associated with each procedural state, and a document may begenerated that includes the images and text near the images thatidentify an action being performed.

The output may include transmitting the electronic data (e.g., inassociation with an identifier of the video data or surgical procedure)to storage or another device. In some instances, the temporalinformation is used to annotate and/or assess video data. For example,it may be determined whether a surgical team traversed through anapproved set of actions. As another example, a video may be annotatedwith captions to identify what steps are being performed (e.g., to trainanother entity).

The electronic data generated using process 1000 may provide a base todescribe surgical consensuses and assess individual proceduralcompliance. Contingent on the ability to define a discrete route througha surgical data structure for a given procedure is the possibility tocompare two separate routes. As individual surgical data structures arecoded, not textual descriptions, process 1000 provides data useful todirectly quantify the differences between two sets of live surgicaldata. The comparison represents the difference in the definingattributes of nodes and actions coded within the two procedures. Furtherto these attributes, the comparison may also include metadata recordedagainst the map such as risk induced during the procedure.

In some embodiments, these data points may include, for exampleinformation relating the surgical data structure such as anatomicalmodifications made not within the original surgical route, time takenfor each step and the types of instrument used, or data retrieved frompatient measurements made during the procedure such as blood loss orloss of heart beat during the procedure.

In certain embodiments, a surgical route is identified based on livesurgical data. The surgical route can then be compared to a pre-computedprocedural profile (e.g., identifying one or more recommended orrequired procedure characteristics. A metric of procedural compliancemay be generated. This metric of procedural guidance represents aquantitative analysis of how well a canonical procedure method wasfollowed. In some embodiments, the metric may be used for training oreducation purposes. The metric may be generated using an “ideal”surgical data structure, which can be used to provide a proceduralcompliance score.

In some embodiments generating electronic data associated withprocedural states enables auto-editing of surgical video and othermaterial to streamline the design of accurate training tools. Usingprocedural step recognition 1004, videos may be annotated, tagged oredited to become more easily navigable as a training tool. Furthermore,this functionality may be used to efficiently reduce the volume of dataneeded to be stored to contain all the crucial steps of surgery.Currently the process of surgical video editing is highly laborintensive and requires manual annotation or editing to produce aneducational surgical video.

In additional embodiments, a defined surgical data structure route maybe used to produce other training tools. By definition, a surgical datastructure holds all the information needed to reproduce a surgicalprocedure, including a detailed anatomical reference model. Therefore,the data held in the specific sequence of particular actions and nodesreferenced within this route may be reproduced or simulated in video, anAR format, or a VR format to produce comprehensive surgical simulationsor animations. Further, these training simulation tools may be adjustedto provide a pre-operative training tool. By integrating patientspecific data to a generic 4D surgical map or surgical data structure ofa procedure, the resulting simulation may be used to refresh a surgeonor other entity involved in a surgical procedure of the preferredsurgical method prior to a procedure.

FIG. 11 shows a process 1100 for updating the surgical data structure.At block 1102, a surgical data structure is accessed. In someembodiments, the surgical data structure is similar to the examplesdiscussed in relation to FIGS. 2, 3, and 4.

At block 1104, a new node is generated for the surgical data structure.A new node may be generated in response to detecting that anode-creation condition was satisfied. The node-creation condition maybe configured to be satisfied upon detecting input that corresponds toan instruction to add a node to the surgical data structure (e.g., withan identification of a corresponding state and/or surgical action). Anupdate to the node-creation condition can be initiated pre-surgery,during surgery, or post-surgery by (for example) an automated processand/or a manual input from a user accessing the surgical data structure.

In some embodiments, an update to the node-creation condition can beinitiated by an automated process. A local server, a remote server, oranother computing device can include instructions that configure thedevice to execute an automated process to update the node-creationcondition. The automated process to update the node-creation conditioncan access surgical data from a single procedure or a dataset thatincludes surgical data from multiple procedures. The surgical data caninclude any of the types of data discussed in relation to block 1002 ofFIG. 10. The automated process to update the node-creation condition canprocess the surgical data into one or more discrete data points. Thediscrete data points can be associated with video data, audio data,representations of inputs received at a device (e.g., a voice command, amotion command or input received at a keyboard or other interfacecomponent), data from smart surgical tools, and so on. The discrete datapoints may be identified from the surgical data using a featureextraction algorithm. In some embodiments, the automated process toupdate the node-creation condition may receive the discrete data pointsfrom another process associated with providing surgical guidance.

After processing the surgical data into one or more discrete datapoints, the automated process to update the node-creation condition maydetermine that one or more of the discrete data points do not correspondto a procedural state associated with a node in the surgical datastructure. If the discrete data points do not correspond to a node inthe surgical data structure, the automated process to update thenode-creation condition may configure the node-creation condition to besatisfied. In some embodiments, the process may be configured totransmit a request for approval to trigger the node-creation conditionto an appropriate approval authority prior to updating the node-creationcondition.

In some embodiments, the node-creation condition may be satisfiedmanually. A user (e.g., surgeon, nurse or team member) may accesssurgical data using a surgical data structure explorer. The surgicaldata structure explorer can be configured to provide a user access tosurgical data associated with a surgical data structure. The user mayprovide inputs to the surgical data structure explorer that satisfy thenode-creation condition. The user inputs can include, for example, a newprocedural state, discrete data points associated with the surgicaldata, and/or additional information that the process can use to generatea new node.

After the node creation condition is satisfied by the automated processor manually, a new node can be generated for the surgical datastructure. The new node will be associated with new procedural metadata.The procedural metadata can include any of the data discussed earlier inthe specification. The process can identify procedural metadata to beassociated with the new node from the one or more discrete data pointsextracted from the surgical data. The process can also receiveprocedural metadata from a user's manual inputs.

At block 1106, the new node can be associated with a new proceduralstate. The generation of the new node and/or association with the newprocedural state may be based on (for example) one or more of userinputs, sensor data from a device (e.g., wearable or non-wearabledevice) located in an operating room, data from a smart surgical tool,and/or data from a camera recording video data in an operating room. Forexample, a set of instructions, executable on the local server, remoteserver, and/or a computing device, can include instructions to updatethe surgical data structure to include a new procedural state associatedwith the one or more discrete data points extracted from the surgicaldata. Further instructions may update the surgical data structure toassociate the new node generated in at block 1104 with the newprocedural state. A computing device accessing the updated surgical datastructure can identify the new node by determining received surgicaldata includes discrete data points that match discrete data pointsassociated with the new procedural state.

The new node generated at block 1104 can be integrated into the surgicaldata structure. At block 1108, a first node and second node of thesurgical data structure 1108 that are related to the new node areidentified. The first node can include one representing a proceduralstate preceding the new procedural state (e.g., corresponding to a timebefore a particular action was performed). The second node can includeone representing a procedural state occurring after the new proceduralstate (e.g., corresponding to a time after a different particular actionis performed). It will be appreciated that, in some instances, more thanone first node and/or more than one second node is identified.

In some embodiments, the related nodes can be determined using theassociated procedural metadata. In some instances, only a subset of thevariables in the procedural metadata associated with a node may includevalues different from a second node in the surgical data structure. Arelation threshold may be set to identify a minimum number of identicalvariables between a first set of procedural metadata and a second set ofprocedural metadata. If the number of identical variables exceeds therelation threshold, the first node may be related to the second node.

At block 1110, a plurality of edges are generated to connect the newnode to the first node and to the second node. In instances where morethan one first node and/or more than one second node is identified, oneor more additional edges can be generated. Each edge may correspond to asurgical action that results in a procedural state transitioning betweenstates represented by the connected nodes. The discrete data pointsextracted from the surgical data can be used to identify a surgicalaction that is associated with transitioning from a first proceduralstate to a second procedural state. Also, procedural metadata associatedwith the new node, the first node, and the second node can be processedto identify surgical actions associated with transitioning from a firstprocedural state to a second procedural state.

At block 1112, the surgical data structure with the new node and the newedges can be updated by the local server, the remote server, and/or acomputing device. For example, an array, relational database or othertype of data structure may be updated to include an elementcorresponding to the new node and multiple edge elements correspondingto the generated edges. The updated surgical data structure can betransmitted to another network connected device.

It will be appreciated that the steps described in FIG. 11 may beperformed during a surgical procedure (e.g., at a local surgery in anoperating room or facility at which the procedure is being performed orat a remote server, such as a cloud server) or after the procedure usingstored data collected during the procedure. For example, in oneinstance, surgical data is collected during a surgical procedure bydevices located in an operating room, and subsequently a remote servermay receive and process the surgical data. As another example, a localdevice may receive and process the surgery data during the procedure.

Specific details are given in the above description to provide athorough understanding of the embodiments. However, it is understoodthat the embodiments can be practiced without these specific details.For example, circuits can be shown in block diagrams in order not toobscure the embodiments in unnecessary detail. In other instances,well-known circuits, processes, algorithms, structures, and techniquescan be shown without unnecessary detail in order to avoid obscuring theembodiments.

Also, it is noted that the embodiments can be described as a processwhich is depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart can describethe operations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations can be re-arranged. A process is terminated when itsoperations are completed, but can have additional steps not included inthe figure. A process can correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination corresponds to a return of the functionto the calling function or the main function.

Furthermore, embodiments can be implemented by (for example) hardware,software, scripting languages, firmware, middleware, microcode, hardwaredescription languages, and/or any combination thereof. When implementedin software, firmware, middleware, scripting language, and/or microcode,the program code or code segments to perform the necessary tasks can bestored in a machine readable medium such as a storage medium.

For a firmware and/or software implementation, the methodologies can beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine-readable mediumtangibly embodying instructions can be used in implementing themethodologies described herein. For example, software codes can bestored in a memory. Memory can be implemented within the processor orexternal to the processor. As used herein the term “memory” refers toany type of long term, short term, volatile, nonvolatile, or otherstorage medium and is not to be limited to any particular type of memoryor number of memories, or type of media upon which memory is stored.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the disclosure.

What is claimed is:
 1. A method comprising: accessing a global surgicaldataset comprising a plurality of surgical data structures includingsurgical data collected during performance of a surgical procedure,wherein each surgical data structure includes: a plurality of nodes,each of the plurality of nodes being associated with one of a pluralityof discrete procedural states for the surgical procedure and with a setof procedural metadata; and a plurality of edges, each edge of theplurality of edges connecting two nodes of the plurality of nodes,wherein each edge of the plurality of edges is associated with asurgical action or surgical event; receiving, by one or more dataprocessors, surgical data for the surgical procedure; detecting, by theone or more data processors, a component from a first portion of thesurgical data; determining a surgical data structure from the pluralityof surgical data structures based on the component from the firstportion of the surgical data, wherein the component associated with thesurgical data structure is determined by processing the surgical data;identifying a node from the plurality of nodes of the surgical datastructure based on the component from the surgical data; retrieving, bythe one or more data processors, from the surgical data structure, a setof procedural metadata associated with the node; selecting, by the oneor more data processors, an edge from among a set of edges, wherein eachedge of the set of edges connects the node to a corresponding next node;generating, based on the selected edge, electronic data to be associatedwith the first portion of the surgical data; and outputting theelectronic data, the output associating the electronic data with thefirst portion of the surgical data.
 2. The method of claim 1, furthercomprising detecting, by the one or more data processors, one or moreadditional components from a second portion of the surgical data;determining whether the one or more additional components in the secondportion of the surgical data corresponds to the surgical data structureor another surgical data structure of the plurality of surgical datastructures; and selecting, by the one or more data processors, thesurgical data structure or another surgical data structure of theplurality of surgical data structures that includes nodes thatcorrespond to the one or more additional components.
 3. The method ofclaim 2, further comprising: transitioning from the surgical datastructure or another surgical data structure of the plurality ofsurgical data structures if the surgical data structure does not includeany nodes corresponding to the one or more additional components;identifying a new node from the plurality of nodes of the anothersurgical data structure based on the one or more additional components;retrieving, by the one or more data processors, from the anothersurgical data structure, a set of procedural metadata associated withthe new node; and selecting, by the one or more data processors, an edgefrom among a set of edges, wherein each edge of the set of edgesconnects the new node to a corresponding next node.
 4. The method ofclaim 1, further comprising identifying, for each edge of the set ofedges, a weight; and identifying, for each edge of the set of edges andbased on the weights, a baseline edge representing a preferred surgicalaction or surgical event, wherein the electronic data indicates whetherthe surgical action corresponding to the identified edge differs fromthe preferred surgical action or surgical event.
 5. The method of claim4, further comprising generating, based on the selected edge, electronicdata to be associated with each data structure of the plurality ofsurgical data structures; and outputting the electronic data, the outputassociating the electronic data with each data structure of theplurality of surgical data structures.
 6. The method of claim 1, whereinthe component associated with the surgical data structure is determinedby processing the surgical data using machine learning.
 7. The method ofclaim 1, wherein the surgical data is live surgical data, the livesurgical data having been collected by a plurality of devices in anetwork involved in performance of the surgical procedure.
 8. The methodof claim 1, wherein outputting the electronic data includes transmittingthe electronic data to another device.
 9. A system comprising: one ormore data processors; and a non-transitory computer readable storagemedium containing instructions which when executed on the one or moredata processors, cause the one or more data processors to performactions including: accessing a global surgical dataset comprising aplurality of surgical data structures including surgical data collectedduring performance of a surgical procedure, wherein each surgical datastructure includes: a plurality of nodes, each of the plurality of nodesbeing associated with one of a plurality of discrete procedural statesfor the surgical procedure and with a set of procedural metadata; and aplurality of edges, each edge of the plurality of edges connecting twonodes of the plurality of nodes, wherein each edge of the plurality ofedges is associated with a surgical action or surgical event; receiving,by the one or more data processors, surgical data for the surgicalprocedure; detecting, by the one or more data processors, a componentfrom a first portion of the surgical data; determining a surgical datastructure from the plurality of surgical data structures based on thecomponent from the first portion of the surgical data, wherein thecomponent associated with the surgical data structure is determined byprocessing the surgical data; identifying a node from the plurality ofnodes of the surgical data structure based on the component from thesurgical data; retrieving, by the one or more data processors, from thesurgical data structure, a set of procedural metadata associated withthe node; selecting, by the one or more data processors, an edge fromamong a set of edges, wherein each edge of the set of edges connects thenode to a corresponding next node; generating, based on the selectededge, electronic data to be associated with the first portion of thesurgical data; and outputting the electronic data, the outputassociating the electronic data with the first portion of the surgicaldata.
 10. The system of claim 9, further comprising detecting, by theone or more data processors, one or more additional components from asecond portion of the surgical data; determining whether the one or moreadditional components in the second portion of the surgical datacorresponds to the surgical data structure or another surgical datastructure of the plurality of surgical data structures; and selecting,by the one or more data processors, the surgical data structure oranother surgical data structure of the plurality of surgical datastructures that includes nodes that correspond to the one or moreadditional components.
 11. The system of claim 10, further comprising:transitioning from the surgical data structure or another surgical datastructure of the plurality of surgical data structures if the surgicaldata structure does not include any nodes corresponding to the one ormore additional components; identifying a new node from the plurality ofnodes of the another surgical data structure based on the one or moreadditional components; retrieving, by the one or more data processors,from the another surgical data structure, a set of procedural metadataassociated with the new node; and selecting, by the one or more dataprocessors, an edge from among a set of edges, wherein each edge of theset of edges connects the new node to a corresponding next node.
 12. Thesystem of claim 9, further comprising identifying, for each edge of theset of edges, a weight; and identifying, for each edge of the set ofedges and based on the weights, a baseline edge representing a preferredsurgical action or surgical event, wherein the electronic data indicateswhether the surgical action corresponding to the identified edge differsfrom the preferred surgical action or surgical event.
 13. The system ofclaim 12, further comprising generating, based on the selected edge,electronic data to be associated with each data structure of theplurality of surgical data structures; and outputting the electronicdata, the output associating the electronic data with each datastructure of the plurality of surgical data structures.
 14. The systemof claim 9, wherein the component associated with the surgical datastructure is determined by processing the surgical data using machinelearning.
 15. The system of claim 9, wherein the surgical data is livesurgical data, the live surgical data having been collected by aplurality of devices in a network involved in performance of thesurgical procedure.
 16. The system of claim 9, wherein outputting theelectronic data includes transmitting the electronic data to anotherdevice.
 17. A computer-program product tangibly embodied in anon-transitory machine-readable storage medium, including instructionsconfigured to cause one or more data processors to perform actionsincluding: accessing a global surgical dataset comprising a plurality ofsurgical data structures including surgical data collected duringperformance of a surgical procedure, wherein each surgical datastructure includes: a plurality of nodes, each of the plurality of nodesbeing associated with one of a plurality of discrete procedural statesfor the surgical procedure and with a set of procedural metadata; and aplurality of edges, each edge of the plurality of edges connecting twonodes of the plurality of nodes, wherein each edge of the plurality ofedges is associated with a surgical action or surgical event; receiving,by the one or more data processors, surgical data for the surgicalprocedure; detecting, by the one or more data processors, a componentfrom a first portion of the surgical data; determining a surgical datastructure from the plurality of surgical data structures based on thecomponent from the first portion of the surgical data, wherein thecomponent associated with the surgical data structure is determined byprocessing the surgical data; identifying a node from the plurality ofnodes of the surgical data structure based on the component from thesurgical data; retrieving, by the one or more data processors, from thesurgical data structure, a set of procedural metadata associated withthe node; selecting, by the one or more data processors, an edge fromamong a set of edges, wherein each edge of the set of edges connects thenode to a corresponding next node; generating, based on the selectededge, electronic data to be associated with the first portion of thesurgical data; and outputting the electronic data, the outputassociating the electronic data with the first portion of the surgicaldata.
 18. The computer-program product of claim 17, further comprisingdetecting, by the one or more data processors, one or more additionalcomponents from a second portion of the surgical data; determiningwhether the one or more additional components in the second portion ofthe surgical data corresponds to the surgical data structure or anothersurgical data structure of the plurality of surgical data structures;and selecting, by the one or more data processors, the surgical datastructure or another surgical data structure of the plurality ofsurgical data structures that includes nodes that correspond to the oneor more additional components.
 19. The computer-program product of claim18, further comprising: transitioning from the surgical data structureor another surgical data structure of the plurality of surgical datastructures if the surgical data structure does not include any nodescorresponding to the one or more additional components; identifying anew node from the plurality of nodes of the another surgical datastructure based on the one or more additional components; retrieving, bythe one or more data processors, from the another surgical datastructure, a set of procedural metadata associated with the new node;and selecting, by the one or more data processors, an edge from among aset of edges, wherein each edge of the set of edges connects the newnode to a corresponding next node.
 20. The computer-program product ofclaim 17, further comprising identifying, for each edge of the set ofedges, a weight; and identifying, for each edge of the set of edges andbased on the weights, a baseline edge representing a preferred surgicalaction or surgical event, wherein the electronic data indicates whetherthe surgical action corresponding to the identified edge differs fromthe preferred surgical action or surgical event.